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Abstract 

This paper is a theoretical study of the following Non-Rigid Structure from Mo¬ 
tion problem. What can be computed from a monocular view of a parametrically 
deforming set of points? We treat various variations of this problem for affine and 
polynomial deformations with calibrated and uncalibrated cameras. We show that in 
general at least three images with quasi-identical two deformations are needed in or¬ 
der to have a finite set of solutions of the points’ structure and calculate some simple 
examples. 


1 Introduction 

Non-Rigid Structure from Motion (NRSfM) from a monocular camera has been addressed 
in a number of papers. The setup is a single camera (monocular) tracking a deforming 
object. Of course if there are no constraints every point in every image has an unbounded 
depth ambiguity. In the literature various constraints have been added, locally or globally, 
to facilitate the computation of numerical solutions of the 3D shape of a deforming object. 

As the general problem is very unconstrained there have been many papers addressing 
certain specifications of the general case, for example instead of a perspective camera 
a weak perspective or affine camera is assumed, Ifll l6l [151 3] [2J or instead of a general 
deformation of the object the assumption is that the nonrigid object deforms as a linear 
combination of K rigid shapes lITOTl . There are also a number of papers that constrain the 
deformation of the object to a physical model or a parameterized family of deformations 
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which they then attempt to solve for in an optimization framework Ifl4l . A review of much 
of the relevant literature can be found in lf2Tj . 

The general case of deforming configurations of points has also received attention, but 
with some restrictions. Some authors consider configurations of points moving with con¬ 
strained motions [26,18]. Other papers treat general motion but restrict their analysis to 
the case of a single point ll22l[T6l[T71l . There has not been much published on the theoreti¬ 
cal underpinnings of the recovery of structure of deforming configurations of points. 

In this paper we analyze, for the first time, the complexity and ambiguities of a fixed 
perspective camera tracking a parametrically set of deforming body of points in 3D. When 
the points move rigidly this is the classic Structure from Motion problem ifTTl . 

The specific deformations we analyze are affine and more generally polynomial. We 
show that when the camera is calibrated and the body undergoes an affine deformation, 
a matching constraint similar to the classical epipolar geometry can be formulated. We 
show that from two images only, one cannot recover neither the deformation nor the orig¬ 
inal points. When three images (i.e. two deformations) are available, we show that in a 
generic situation, the remaining ambiguity is still three-dimensional. However, when the 
two deformations are quasi-identical (see below for a complete definition), we show that 
it is exactly one answer. 

Also we show that an invariant shape description can be recovered from 3 images, i.e. 
from a first view and two other views after deformations. The recovery of this invariant 
does not require calibration. 

Then we turn our attention to the case of complete reconstruction (deformation and 
structure) for general polynomial deformations. We show that it can be calculated, from 
a calibrated camera and 3 images, i.e. from the first view and a two other images coming 
from the same deformation repeated twice. We also show that the number of points needed 
to be tracked depends on the number of parameters of the deformation. 

The case of a full 3D projective transformations is not doable as the deformation can 
always be explained by an affine tansformation; 



We are mainly interested in the theoretical possibilities both as to number of corre¬ 
sponding points and number of images needed. We present algebraic solutions and do not 
treat the numerics and instabilities of these problems. 


2 Affine Deformations 


We start with the study of point correspondences under affine deformations. This has both 
a practical and a theoretical impact. On the practical side, we shall see that one can write 
the matching constraint in a similar manner to the classical epipolar geometry. Thus, find¬ 
ing correspondences between two images of an affinely deforming configuration of points 
can essentially be done with the same machinery as that for the case of classical images of 
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rigidly moving bodies. Of course, we consider only invertible affine deformations. This is 
choice is motivated to avoid degenerated situations, where two distincts points collapses 
to a single points after the deformation. 

On the theoretical front, we shall use this result extensively in the sequel. 


2.1 Essential Matrix 


Let us consider a set of deforming points being imaged with a calibrated camera, which 
can be assumed to be [/; 0]. The projection of a point P in homogeneous coordinates in 
the first image is in homogeneous coordinates [/; 0]P, while the projection in the second 

/a t\ 

image is in homogeneous coordinates [/; 0] ( 1 P. Eliminating P from these two sets 


of equations leads to a bilinear constraint over the corresponding image points q, q', that 
can be written through a so-called essential matrix E : q"Eq = 0. 


Lemma 1. The essential matrix of this pair of images is: E = [t\ x A, where |/:] x is the 
matrix of the cross product with t in the standard basis ofM 3 . 

Proof Let us consider a point P in P 3 not at infinity and projected to q in the first image. 
Then q = [/; 0]P. Thus P = [Ag, If, for some A 6 1. It is projected in to the second 
image as q' = [A;t\P. Thus of = (A Aq + t ). Then [t] x q' = [t] x Aq, where [t\ x is the 
matrix of a t-A t A a, in the standard basis of M 3 , with A being the cross product. Since 
[t] x is a skew-symmetric matrix, this yields q' t [t] x Aq = 0. Thus E = [t] x A. □ 


We denote by = equality modulo multiplication by a non-zero scalar. 

If there are matching pairs of points between the two images: (g i; g')i=i,-,n, the fol¬ 
lowing equations hold: 

q'fEqi = 0 (1) 

for each i and rank E is 2 as rank |/.] x is 2. Indeed the deformation is assumed to be 
invertibel, i.e. det(Al) f 0. E can thus be computed from 7 pairs of corresponding points 
in general position or linearly from at least 8 pairs of points. 


2.2 Deformation Recovery 

Once the essential matrix E = \t] x A is computed, deformation can be recovered up to a 
4-parameter ambiguity. To show this, we recall a lemma from IflTl (page 255): If a rank 
2 matrix F can be decomposed in two different ways as F = [t] x A = [t] x A then there 
exists a constant A 7 ^ 0 and v 6 M 3 , such that: t = Xt and A A = A + tv 1 . 

Can more than two images help? Let us consider the situation where the deformation 
between the first and the second image is ( A , a) and between the second and the third 
(B,b). We have now three distinct essential matrices: E u = [a] x A, E 2 3 = [b\ x B and 
P 13 = \Ba + b] x BA. 

Lrom E 12 , we can compute a 0 and A 0 such that 3a ^ 0,a = aa 0 and 3vi,A = 
From P 2 3 , we can compute b 0 and B 0 such that 3/3 0,b = /3b 0 and 
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3v 2 , B = jj(B 0 + ^o?4) ■ From the third essential matrix A 13 we can compute c 0 and C 0 
such that: 

37 7 ^ 0, Ba + b = 7 C 0 (2) 

and 

3u 3 , 5A. = —(C '0 + CoU 3 ) (3) 

7 

From equations © and ©, we get the following system: 

a(B 0 + 6 0 t4)a 0 + f3 2 b 0 - fijco 
y(B 0 + b 0 v t 2 )(A 0 + a 0 v \) - a(3(C 0 + c 0 vf) 

Furthermore one has to enforce the that none of a, [5 nor 7 vanishes. Formally, this 
is equivalent to computing in the localization of the polynomial ring with respect to these 
variables 0. Concretely, one must introduce new variables: x,y,z and the equations: 

ax — 1 = (3y — 1 = 72 — 1 = 0 . 

Eventually the number of unknowns is A r = 9 + 3 + 3 = 15 and we have exactly 15 
equations. They define an algebraic sub-variety X of A 15 (see lfl3l f8]Q. where we denote 
by A k the A —dimensional affine space. Now the task is to compute the dimension of X. 
Our data is expressed as real numbers. However nothing prevents us from viewing the 
variety X as defined over the field of complex numbers. This does not alter its algebraic 
dimension (see for instance exercise 3.20 in [|T2lO . Moreover C is an algebraically closed 
field. This allows one to benefit from algebraic geometry theorems. 

Indeed our concern now is to determine the dimension of X. If it is zero, this means 
that X is a finite set. In this case, one would be interested in its degree in order to estimate 
the number of solutions. However if the dimension is positive, there are infinitely many 
solutions. A word of caution is necessary here. While the algebraic dimension is the same 
over the reals or the complex numbers, the topological dimension can varry. When the 
dimension is zero, or in other words when the variety is a finite set, this property remains 
unchanged over reals or the complex numbers. When the dimension is strictly positive, 
it might be that the set of real solutions is finite or even empty. In our setting, we will 
see that when the algebraic dimension is strictly postive, the set of real solutions is also 
infinite. 

We prove that indeed X has strictly positive dimension. 

Theorem 1. For two unrelated deformations, the dimension of X is always greater or 
equal to 3. Moreover for generic deformations, it is precisely 3. 

Proof Let a 0 , /3 0 , 70 , x 0 , y 0 , z 0 , v w , v 20 , v 30 be a point on X. Let a = a 0 a 0 ,A = l/a 0 (A 0 + 
a 0 v{ 0 ), b = /3 0 a 0 , B = 1//3 0 (B 0 + b 0 vl 0 ). Then we know that Ba + b = 7 0 c 0 and 
BA = l/ 7 o(C 0 + Co^o)- Consider the variety V defined by the following system: 


r a'(B + bv'^a + (3 ,2 b - (I'yfBa + b) = 0 

7 '(B + bv 2 )(A + av'i) — a'f3'(BA + (Ba + &)u 3 ) = 0 

< a'x' — 1 = 0 

/?y - 1 = 0 

k 7 V - 1 = 0 


= 0 
= 0 


(4) 
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The varieties X and Y are easily seen to be isomorphic by the linear mapping: 


{a,P,'1,x,y,z,v{,v t 2 ,vl) ^ 

(a', p y, x', y\ z\ <, v$, v'j) = 

(a/a 0 , P/Pq, 7/70, a 0 x, p 0 y : 70 z, 1 /al(v{ - v{ 0 ), 1 / Pl(v\ - i4), l/'yliyl - v\ 0 )) 


Therefore dim(X) = dim(Y'). 

The variety Y depends on the deformations (A, a) and (B, b ). One can define a family 
of varieties, whose fiber at (A, a, B. b ) is precisely the variety Y. To proceed, we shall 
define D = A 12 to be the space of affine deformations. Then consider the variety Z C 
A 15 x (D x D) defined by the above system, where A, a, B, b are also considered as 
variables. Then the projection 7 r 2 : Z —» D x D defines a family of varieties whose 
fibers are precisely the different varieties Y. 

The variety Z is embedded into A 15 x D x D = A 39 and is defined by 15 equations. 
Therefore we have: dim(Z) > 39 — 15 = 24. 

The projection 7 t 2 is surjective as easily seen from the system. For each pair of de¬ 
formations (A, a) and ( B, b), the point defined by a = P = ^/ = x = y = z = l and 
v\ = 17 = v : > = 0 is in the fiber. 

/ -41 -69 83 \ 

Now choose a pair of deformations: s = (A s , a si B s , b s ) with A s = —2 —17 27 , 

\ -97 -95 95 / 

/ 87 \ / 2 -49 61 \ / 39 \ 

a s = I —51 J, B s = I —59 —49 —70 1 and b s = I —73 1. Compute the fiber 
\ -98 / \ -68 -90 6 J \ 62 / 

of 7 t 2 at s: Y s = 7 r^" 1 (s). It turns out that dim(Yg) = 3 and that Y s is a smooth variety. Let 
diT 2 be the differential of 7 t 2 . Then the tangent space of Y s at a smooth point is included 
into the kernel of d7r 2 . Therefore dim(ker(o?7r 2 )) > 3. 

Now a simple computation shows that <Ar 2 is surjective at the point defined by a = P = 

7 = x = y = z = 1, Vi = v 2 = v 3 = 0 and s. At this point as just mentioned, Y s is smooth 
and thus its tangent space is three-dimensional. Therefore the dimension of Z, given by 
dim(ker(d 7 r 2 )) + rank(d 7 r 2 )), is greater or equal to 3 + rank(d 7 r 2 | 7 r -i ( - ;7 - ) ) = 3 + 24 = 27. 

On the other hand, since there exist three-dimensional fibers, the dimension of Z must 
satisfy: dim(Z) <= 3 + 24 = 27 (see GOl . chapter 4). 

All together, we have proved that dim (A) = 27 and that the generic fiber is three- 
dimensional. □ 


In the relation to the discussion above, one can check that in the neighborhood of each 
real solution, there infinitely other reals solutions. For examples provided that (A, a), (B, b) 
is a solution, then (AA, a), (B, b) for A £ IR\{0} is also a solution, since the essential ma¬ 
trices remains unchanged up to a scale. 

The practical consequence of this theorem is that one cannot hope to recover deforma¬ 
tions from three images. 

When the same deformation is repeated twice the system of equations is simplified. 
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But before we proceed more in depth, let us make the following observation. The 
deformations (A, a) and (AA, Aa) for A f 0 produce the same image. Therefore one 
could conclude somewhat hastily that whatever the number of images, one can except just 
to recover the deformation modulo this equivalence. However observe that if multiples 
(A A A a) and (//A //a) of the same deformation are applied consecutively we get the fol¬ 
lowing overall deformation {p\A 2 , pXAa + pa) which is equivalent to ( A 2 , Aa + a) only 
if yuA = /i or equivalently A = 1. Therefore if exactly the same deformation is repeated 
twice, one can hope to be able to fully recover it. This is the conclusion the deeper analysis 
below will exhibit. 

Consider the essential matrices E\ 2 and E \ 3 . E 23 is the same as E l2 . We compute 
Aj, o-o, C 0 , c 0 as previously and for the actual deformation A, a there exist a f 0 , v\ 7 R 3 , 

7 7 ^ 0 and v 3 6 M 3 , such that: 

a = crao 

A = 1 /a{A 0 + a 0 wj) 

Aa + a = 7 Q) 

A 2 = l/ 7 (Cb + cqv\) 

This results in the following system of equation: 

(A) + a 0 v{)ao + aa 0 - 7 C 0 
7 (A + o 0 nj ) 2 - a 2 {Co + c 0 v 3 ) 
ax — 1 

7^ — 1 

Theorem 2. For a generic affine deformation, repeated twice, one can recover this defor¬ 
mation from the three images. Moreover in a generic situation the solution is unique. 

Proof. Here X designates the sub-variety of A 10 defined by the system [5] Let o- 0 - Ah A- A v 30 
be a point on X. Let a = a 0 a 0 , A = l/a 0 {A 0 -t-ao^o). Then we know that Aa + a = 7 0 c 0 
and A 2 = 1 / 7 0 (C 0 + Cov\ Q ). Consider the variety Y defined by the following system: 

{ {A + av'ffia + a'a — 7 '{Aa + a) 
ffi{A + av'ffi 2 — a' 2 {A 2 + {Aa + a)v 3 ) 
a'x' — 1 
7 V - 1 

The varieties X and Y are easily seen to be isomorphic. Indeed the following linear 
mapping: {a, 7 , x, y , z, v\, v\) ^ {a', 7 ', x', z' , v'f v%) = {a/a 0 , 7 / 70 , a 0 x, ffiy, 70 z, l/al{v{- 
v{ 0 ), 1 / 7 2 (A — vf)) is an isomorphism from X and Y. Therefore dim(X) = dim(F). 

As in theorem HI let us consider the variety of all affine deformation D = A 12 , the 
variety Z C A 10 x D defined by the system[ 6 ]and eventually the projection: 7 r 2 : Z —> D. 

Observe that here again the map ir 2 is surjective, since given an affine deformation 
{A, a), the system [ 6 ] always admits a' — 7 ' = x' — y' — 1 and v[ = v' 3 = 0 as solution. 

This implies that dim(Z) > dirn(D) = 12. As known, the dimension of any fiber of 


0 

0 

0 

0 


( 6 ) 


0 

0 

0 

0 


(5) 
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7 r 2 is at least dim(Z) — dim(D) and the minimal dimension of a fiber will exactly be 
dim(Z) — dim(_D). (see 11201 . chapter 4). 


Now consider the deformation s = (A,, a s ) defined by A s 


27 

99 

92 

8 

29 

-31 

69 

44 

67 


and 


-32 


a s 


-74 

-4 


At this point, the fiber is zero dimensional, which shows that dim(Z) = 


12. Therefore for a generic deformation repeated twice, one can recover the deformation 
up to a finite fold ambiguity. 

Now let us show that in a generic situation, there is exactly one answer. Observe that 
this is the case for the particular fiber considered above. Thus let p £ Z be the unique 
point such that n 2 (p) = s. As easily computed, the variety Z is smooth at this point and 
the projection ir 2 is etale at p. Thus there exits U an open neighborhood of p in the Zariski 
topology, such that (i) Z is smooth on U and (ii) the projection 7 t 2 is a diffeomorphism 
from U to V = ir 2 (U). Therefore the map tt 2 \u —> D is flat. Then all the fibers are 
isomorphic to the fiber over s and thus are made of a single point. 

Moreover if there was another component of dimension 12, since the fiber over s is 
smooth, p cannot lie on the intersection of U and such a component. Thus the fiber over 
s should contain more than a single point, which is not the case. All together, we have 
proven that there is a single component of maximal dimension in Z and that the generic 
fiber over D is a singleton, or in other words, that for a generic situation, there is a single 
solution. □ 


Here it is clear that the unique solution is real, since this is exactly the actual deforma¬ 
tion that the points have undergone. 

There are cases, other than two identical transformations, where the deformations are 
also solvable. 

For example, when B = A A and b = pa for unknown, non-zero scalars A, //. The 
system of equations |4] reduces to: 

f A(A 0 + aov\)a 0 + apa 0 - 70) =0 

\ Ay(Ao + a 0 v\) 2 - a 2 (C 0 + cqv\) = 0 ' * 

This system is similar, but still different, than system [5] Of course, as previously, one 
has to add the two further equations ax — 1 = yz — 1 = 0. Now we shall prove the 
following result. 

System [7] defines a discrete variety. As a consequence, 

Theorem 3. If (A, a) is the first deformation and (A A pa) the second deformation fA / 0 
and // f 0), one can recover the two deformations and the structure up to a finite fold 
ambiguity. 

Proof. We proceed as in the the previous theorem. Here X designates the sub-variety of 
A 10 defined by system[7](together with equations ax — 1 = yz — 1 = 0). Note that A, p are 
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not unknowns but parameters. Let a 0 , 70 , x 0 , z 0 , v 10 , v 30 be a point on X. Let a = a 0 a 0 , 
A = l/a 0 (A) + aoWio). Then we know that A Aa+pa = 7 0 c 0 and XA 2 = l/'y 0 (Co + c 0 v^ Q ). 

Consider the variety Y defined by the following system: 

{ A(A + av'l)a + a' pa — Y(XAa + pa ) 

A7 '(A + axf) 2 — a ,2 (AA 2 + (AAa + pa)v%) 
a'x' - 1 
7V - 1 

The varieties Af and V' are isomorphic by the following linear mapping: 

O, 7, x, y, z, v{, vl) t-y («', 7', x', z', <, v$) 

= (a/a 0: 7/70, a 0 x, /%, 7 o-, l/«o( v i ~ ^10), V 7 o( w s - v lo)) 


0 

0 

0 

0 


( 8 ) 


is an isomorphism from X to Y. Therefore dim(Af) = dim(F). 

Again consider the variety D of all pairs ((A, a), (AA, //a)) (with A ^ 0 and // 7 ^ 0) 
where (A, a) is an affine deformation. Then D = A 12 x (A\{0}) 2 . 

Consider also as previously the variety Z C A 10 x D defined by the system [ 8 ] and 
eventually the projection: 7 t 2 : Z — y D. 

Again observe that here again the map n 2 is surjective, since given an affine deforma¬ 
tion (A, a) and two non-zero scalars A, /i, the system [ 8 ] always admits a' = 7 ' = x' = 
y' — 1 and v[ — v' 3 — 0 as solution. This implies that dim(Z) > dim(D) = 14. As 
mentioned above, the dimension of any fiber of 7 r 2 is at least dim(Z) — dim (A) and the 
minimal dimension of a fiber will exactly be dirn(Z) — dim (id). 

" 27 99 92 

Now consider the deformation s = (A s , a s , X s , /J-s) defined by A s = 


69 


29 -31 
44 67 


etc 


which s 


-32 

-74 

-4 


A. s = 1 and // s = 1 . At this point, the fiber is zero dimensional, 


;hows that dim(Z) = 14. Therefore for a generic deformation and a generic point 
(A, fi) G (A\{0}) 2 , one can recover the deformation and the scales A, // up to a finite fold 
ambiguity. 

Here again, let us show that in a generic situation, there is exactly one answer. As 
previously, observe that this is the case for the particular fiber considered above. Thus let 
p G Z be the unique point such that it 2 (p) = s. Again, as easily computed, the variety 
Z is smooth at this point and the projection n 2 is etale at p. Thus there exits U an open 
neighborhood of p in the Zariski topology, such that (i) Z is smooth on U and (ii) the 
projection 7 t 2 is a diffeomorphism from U to V = tt 2 (U). Therefore the map n 2 \u —y D 
is flat. Then all the fibers are isomorphic to the fiber over s and thus are made of a single 
point. 

Moreover if there was another component of dimension 14, since the fiber over s is 
smooth, p cannot lie on the intersection of U and such a component. Thus the fiber over 
s should contain more than a single point, which is not the case. All together, we have 
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proven that there is a single component of maximal dimension in Z and that the generic 
fiber over D is a singleton, or in other words, that for a generic situation, there is a single 
solution. □ 

2.3 Shape Recovery 

Once the deformation is known the shape before and after deformation is easily calculated. 
Indeed from the first image, the points are known up to scalar multiplication. From the 
second image, this scalar for each point is computed by linear means. The complicated 
part is to compute the deformation and this is our focus. 

2.4 Critical Surface 

Are there point configurations that do not allow the recovery of the essential matrix? It 
turns out the situation is similar to the classical case. 

Assume that the projected points before and after deformation do not constrain the 
essential matrix uniquely. Therefore there exists more than one solution (homogeneous) 
to the system: qjEpi = 0. One is the correct solution E 1 = [f] x A, while another solution 
E 2 would have another decomposition. Therefore if there exists another solution, the 
points must satisfy: 

qlEopi = Pi 

This means that the original points in space lie on a quadric, whose equation involves 
the affine motion that we are looking for. In this case, the recovery presents an additional 
layer of ambiguity. There exist several essential matrices and for each essential matrix, the 
corresponding affine motion is recovered up to the ambiguity described above. 

2.5 Invariant Shape 

The shape of a deforming object is by definition changing but there are descriptions that are 
invariant to the transformations, we shall show when these descriptions can be recovered 
from a sequence of images of a deforming object. 

2.5.1 Equations 

Let P 0 ,Pi,P 2 ,P 3 , ...,P n -i be 3d points in homogeneous coordinates with 1 as the last 
coordinate. Here and during all section [23] the points P 0 , Pi, P 2 , P 3 are assumed to define 
a affine basis of the three dimensional affine space A 3 . 

If a point P satisfies P = aP 0 + fiP\ + yP 2 + (1 — a — /3 — 7 )P 3 , if it undergoes a 3D 
affine transformation, T, then, 

TP = aTP 0 + f3TP l + 7 TP 2 + (1 - a -/3 - 7 )TP 3 . 


E 2 [I ; 0]Pj — 0 
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Thus, («,/?, 7) is an affine invariant and a. i3, 7 and 1 — a — f3 — 7 are the affine in¬ 
variant coordinates of P. In this section we aim at computing this affine invariant. The 
transformation itself is not recovered here. We deal with the simultaneous recovery of the 
transformation and the point coordinates in section [3 

The real advantage of this affine invariant is that it does not require camera calibration, 
while full recovery of deformation and structure requires it. On the other hand, the affine 
invariant description only provides structure up to an unknown affine deformation. 

Let us write down the equations for the two image point sets {q t } and {7/'} of an affinely 
(A t\ 

changing point set where I ) is the affine transformation and C the unknown camera 


matrix. 


• For the first image, before the deformation, for each i, we have: q, = G'[a,Po + 
/3iPi + 7 jP 2 + (1 — on — A — 7 OP 3 ]. Thus each image point gives two equations. 

• After the deformation, in the second image, for each i, we have: q[ = C ^ P, = 

C ^ ^ [ajP 0 +A-Pi+ 7 i-P 2 +(l—ccj— Pi— 7i)P 3 ]. Again this yields two equations 
per point. 


Basically the system has 2 x 2n = 4n equations, where n is the number of points. As 
for the unknowns, there are 4 x 3 + 3(n — 4) + 12 + 12 = 3n + 24 unknowns namely 

P 0 , Pi, P 2 , P 3 , {a,, /3 i; 7 i}i> 4 , the camera matrix C and the affine transformation 

There are still ambiguities as, for any full rank V a 4 x 4 matrix with last row [0, 0, 0,1], 

CP = (CV){V~ l P), (new camera x new points) and C P — (C'17)(l /^ 1 V)(V~ l P ), 

(new camera x new affine transformation x new points). Since the only unknowns that are 
relevant are {a*, !3 tl 7 j}j> 4 , we can assume C = 7: 0], removing 12 unknowns. This for¬ 
mally makes the computation identical to the case of a calibrated camera, while calibration 
is not required here. 

To make things more explicit, let us introduce new variables Ao, Ai, A 2 , A 3 , such that 
Pi = A iQi for 0 < i < 3. Then the equations can be written as follows: 



qi A [ai(Ao<?o) + Pi(Xiqi) + 7i(A2<?2) + Si(X 3 q 3 )} = 0 

1 


Qj A [A 7] 


= 0 


Qi A [A; t] 


ai(Xoqo) + Pi{Xiqi) + li{X 2 q 2 ) + Si(X 3 q 3 ) 
1 


= 0 


(9) 

( 10 ) 

( 11 ) 


where 0 < j < 3 and 4 < i < n — 1 and Si = 1 — ctj — — 7 $, A being the cross product. 

These equations are homogeneous in {a lv 4 }i<ij,fc <3 where A = [a tJ ] and t = [f 1; t 2 , f 3 ]*. 
Therefore they define an algebraic variety X in P 11 x A 4 x A 3(n_4i (see [|T3l HI), where 
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A k denotes the k —dimensional affine space. More precisely, since none of {A 0 , Ai, A 2 , A 3 } 
should be zero, we need to compute in the localization of the polynomial ring with respect 
to each A* Q. Here again, this is done by adding new variables {w 0 , u 3 ,u 2 , u 3 } and the 
equations: 

\i ■ Ui — 1 = 0 ( 12 ) 

Putting together all the equations (191101111121) we get a variety X C P 11 x A 8 x 
A 3 ( n —4). Since (A 0 , Ai, A 2 , A 3 , u 0 , Mi, u 2 , u 3 } and {a^, t k }i<ij,k <3 are not of interest, we 
eliminate them from the system and get a system involving only {«;, if, j,} 41 - This 
is equivalent to projecting A" over A 3(n ~ 4j , we are concerned with the case n > 5. The 
question now is: does this define a zero-dimensional variety or in other words can the affine 
invariant describing each point be computed up to a finite fold ambiguity. We address this 
question in the following subsection. 

2.5.2 Dimension Analysis 

The case of two images Let X be the closure of X in P 11 x P 8 x p 3 ( n “ 4 ). Let us denote 
7T! : X —* P 11 the projection on the first space. The image Y) of X by 7Ti is precisely the 
variety of affine deformations that can explain the two images. If we have enough points 
n > 7, then by the beginning of section lA2l its dimension is exactly 4. Moreover it is 
irreducible being isomorphic to A 3 x A\{0}. 

Let us determine what are the fibers of Given the deformation {A, t), from equa¬ 
tions (flOl) . one can compute Xj for j = 0,1,2, 3. Then from equations © and (fill) , we can 
compute {a^ Ti} 4 <i<n-i- Therefore, the fibers are all finite, such that X is irreducible 

and dim(A) = 4. This can be rephrased very easily by saying that given the deformation, 
one can indeed recover the structure! 

Now consider 7 t 3 : X —> p 3 ( n ~ 4 ) j the projection on the third space. Let Y 3 be the 
image tt 3 (X). In order to answer the initial question: can the affine coordinates can be 
recovered from two images or equivalently one deformation, one need to determine if 
Y 3 is a discrete variety. In case it is, we could conclude that affine coordinates can be 
computed from two images up to a finite fold ambiguity. 

We have that Y 3 is a discrete variety if and only if, the generic fiber of 7 t 3 has at least 
dimension 4. However for a generic situation, this is not the case as explained below. 
Indeed, given {a t , Pi, 7 i} 4 <t<n-i> equations © enable the computation of A 0 , A 3 , A 2 , A 3 , 
up to a scale factor. Similarly equations (flOl) and (fill) enable in a generic situation, the 
recovery of [A; £] up to a scale factor if n > 7. Therefore the generic fiber of 7 t 3 is bi- 
dimensional, thus Y 3 cannot be a discrete variety. This yields the following conclusion: 

Theorem 4. Two images are not enough to recover the affine invariant coordinates of the 
points Pi for 4 < i < n — 1. 

One can wonder if some prior knowledge of the world can help to get a finite set of 
solutions. We show the following theorem: 
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Theorem 5. If the 4 points {P 0 . P { , I\, P 3 } are known and n > 8, then one can compute 
the affine invariant coordinates of the other points {Pi}i<i< n -i from two images from a 
single non-calibrated camera, if the scene undergoes a general affine deformation and the 
points are in generic position. 

Proof Since, n > 8 and the points are in generic position, one can compute the es¬ 
sential matrix, from the affine deformation can be extract up to a four fold ambiguity. 
The parametric representation of the set of acceptable deformations can be plugged in 
equations (fTOl) . Since the variables {A 0 , Ai, A 2 , A 3 } are known, this allows to completely 
recover the deformation. 

This situation can be described more formally as follows. Let Z be the variety in 
P 11 x A 3(n_4 ) defined by equations (1911011 II) . Note that in this case, we do not consider 
{Ao, Ai, A 2 , A 3 } as variables, since they are known and thus we do not include the local¬ 
ization equations (fl2l) into our system. 

Let 7r i : P 11 x A 3(n ~L be the projection on the first factor. From the previous discus¬ 
sion, the closure tt\ (Z) is a finite set. The fiber is also finite as mentioned above. Therefore 
dim (A) = 0, that is Z is a finite too. 

Therefore the projection of Z on the second of factor is necessary a finite variety, 
meaning that the affine invariant coordinate can be recovered up to a finite fold ambiguity. 

□ 

In order to illustrate this approach, we proceeded to the following experiments. Con¬ 
sider the points P 0 = [0, 0,1,1]*, Pi = [1, 0, 0,1]*, P 2 = [0,1, 0,1]*, P 3 = [1,2,1,1]*, 
P 4 = [1, -1, -1,1]*, P 5 = [-2, 3,1,1]*, P 6 = [2, -2, 3,1]*, P 7 = [-4, -2,1,1]*, P 8 = 
[3, 2, 3,1]*. The points are projected onto the first image to produce {gj}o<i<8- The vari¬ 
ables A 0 , Ai, A 2 , A 3 are assigned their true values. The system of equations is formed and 
the deformation is eliminated, resulting in a system over {«*, 7 i} 4 <i< 8 > which turns out 

to define a discrete variety. The computations were performed with Maple 16. 

Three images In this section, we investigate the case of three images. Two configura¬ 
tions are possible, either the same deformation is repeated twice or two different deforma¬ 
tions are performed. 

The case of two distinct deformations is quickly dealt with, relying on theorem [Q one 
can prove the following result: 

Theorem 6. When the points undergo two unrelated generic affine deformations, the affine 
invariants ( a i: , 6,, yf) cannot be computed up to a finite fold ambiguity. 

Proof. If we stack equations ©, (fTOl) . (UTl) and (IT2l) . the equations coming from a third 
image generated by a generic distinct affine deformations, we get a variety X embedded 
in P 11 x P 11 x A 8 x A 3 ( n-4 ). Projecting this variety over P 11 x P 11 , we get the variety 
defined by the essential matrices. As known from theorem [H this variety has strictly 
positive dimension. Then dirn(X) > 0 and the projection over A 3 '"~ 4) will also have 
strictly positive dimension, since the generic fiber of this second projection is finite. □ 
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Therefore let us now turn our attention to the case where the points undergo the same 
deformations twice. In this context the unknowns are exactly the same as in the two 
images case: {A 0 , Ai, A 2 , A 3 }, {<%, f fc }i<ij,fe< 3 > t = [A, £ 2 ,^ 3 ]* and (a*, 7*}4<i<n-i- 

The equations involved in this situation also contain those of the two images case and in 
addition equations similar to (fTOl) and (ITTI) . Finally, with P* = aj(A 0 go) + A(Ai<Zi) + 
7 *^ 292 ) + 5i(A 3 g 3 ), we get: 


qi A Pi 


q'j a [A; t] 


l 


q[ A [A; t] 



q'j A [A 2 ; At + t] 
q'i A [A 2 ; At + t] 


x jqj 

1 

'Pi 

1 


0 

0 

0 

0 

0 


( 13 ) 

(14) 

( 15 ) 

( 16 ) 

(17) 


As above, we add to these equations, the localization constraints expressed in [12l All 
together we get a variety X C P 11 x A 8 x A 3(n ~ 4) . Again, we are interested in the 
projection of these variety into the factor A 3( ”~A. However here we are in a position to 
prove the following result. 


Theorem 7. If the points undergo the same deformation twice, one can compute the affine 
invariant structure, i.e. {a*, 7 i} 4 <i<n-i up to a finite fold ambiguity from the three 

images. 


Proof The proof is quite clear and works with the same scheme as the previous proofs. 
By eliminating from the equations the variables other than the affine deformations, we 
get exactly the same equations of the essential matrices. From theorem [2] we know that 
there is a single solution for the affine deformation. Then the other variables are uniquely 
determined. As a consequence, dim(X) = 0. Therefore if one first eliminates the variables 
related to the deformation and the A*, a discrete variety for the affine invariant (a,;, yf) 
is left. □ 


While this results provides a theoretical understanding, practically the task is quite 
cumbersome. One has to eliminate the following variables (A, t , A 0 , A 3 , A 2 , A 3 , u 0 , Ui,u 2 , u 3 ) 
from the system made of previous equations, which leads to a system on {a*, 7 i} 4 <i< n -i 

only. This latter system defines a discrete variety that can be theoretically computed. 


3 Polynomial Deformations 

In this section, we consider general polynomial deformations, the affine transformations 
treated above are a particular case. As previously, we shall assume that no degeneracies 
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occur, i.e. two points never collapse to a single point. This is ensured by assuming that the 
polynomial map is injective. Since we regard the deformation as a polynomial map from 
the complex three-dimensional affine space to itself, it is equivalent to say that it bijective. 
This is known as the AxGrothendieck theorem. Of course, as previously the map itself has 
real coefficients in concrete situations. 

3.1 Universal Matching Constraints 

The essential matrix introduced in section [2711 relies on the elimination of the 3D points 
from the projection equations before and after the deformation. For general polynomial 
deformations, such a process can also be carried out, leading to a polynomial matching 
constraint rather than a bilinear constraint. 

In order to make things more concrete let us examine the equations. Let {qi}i be the 
first image points. Let $ be the deformation that the points undergo. Then in the second 
image, we get the following points: q[ = $(Pj), where P t = a^. 

$(X, Y, Z) = (MX, Y, z ), MX, Y, Z ), MX, Y, Z)), 

where M M 4>3 G M[X, Y, Z] are polynomials. The system of equations is: 

OLi ■ Ui — 1 = 0 ( 18 ) 

q'i A $(aaqi) = 0 (19) 

Here i indexes the points. Equations (fl8l) are again localization equations, expressing that 
none of the a, should be zero. 

The matching polynomial is then obtained through the elimination of a and u from the 
3 polynomial equations: 


<lx®3 (aq) = <&i(aq) 

q y $ 3 (aq) = $ 2 (ag) 

cm = 1 

where all the unknown parameters of the deformation, q and <{ are treated as variables. 
The elimination polynomial is treated as a k term polynomial in the variables of q and q' 
and can be recovered linearly using k point matches. 

For example, if $ is of the following form: 

= o x X + b x Y + c x Z + d x 

$2 = a yY^ + byX" + CyZ 2 

$ 3 = a z Z 3 

where a x) b x ,c x ,d x ,a y ,by,c y , a z are unknown parameters then the elimination polynomial 
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is: 


o 2 r 2 / 4 . 3 / 6 /2 /2 7 2 /2 73 /2 

3 a ybyq x QxQy H~ UyQxQy ~ ^z^x^yQy — Q'xQ'zCyQxQy — a z b y c x q x q y — c^x^zbyq x q y ~ 


a z b x c y q y q'y - a z b x b y q 2 x q y q' 2 - a y a z c x q 2 y q' 2 - a x a y a z q x q 2 y q y - a y a z b x q\(l 2 - a 2 d x q\ 


which is a linear combination of the following 20 terms 


QxQx: qWxffyi qWx, 9 x 9 ?, 


qIq'xQ^ dlq'xdr vlw'yi 9x0*> qlq 


/2 
2/ ’ 


2 /2 /2 / 6 / 4 3 /2 / 2 2 /2 /2 / /3 

QxQyQy , % , g a 5 v , q x q y , %% , q x q y , %% , % , q x , % , 


9 


/2 

y 


so the matching polynomial can be found with 20 matching points from 2 images. 


3.2 Model Selection 

The universal matching constraint exhibited in the previous section depends on the de¬ 
gree of the deformation. More precisely the number of monomials in qi and q[ for each 
matching pair (q i: g') is a function of the deformation degree and also of further assump¬ 
tions on the deformation if there are available, as in the example above. These monomials 
must satisfies a linear equation: they form together the coordinates of points in a higher 
dimensional space. 

Then computing the matching constraints is equivalent to finding the hyperplane that 
best fit all these points. This observation leads to a straight forward algorithm for model 
selection. A model, defined by the degree of the deformations and further assumption 
on the form of deformations if available, is rejected in the hyperplane assumption in the 
higher space is not acceptable. This can be implemented using some information criterion. 
One can refer for instance to lf24l for further details. 


3.3 Dimension Analysis 

As we shall see again, at least three images are required. The camera is assumed to be 
calibrated described by the matrix [/; 0]. 

We shall introduce a few notations. Let us denote T d the variety of deformations of 
degree d. It is a mere affine space. Of course the following canonical injection holds: 

C X d for k < d. Also let 9Jtfc(q, q' j be the subvariety of X/,, defined by the matching 
constraints from the input q = (gi, • • • ,q p ) and q' = (q[, ■ ■ ■ , q' p ) where p is the number 
of terms appearing in the matching constraints. Of course p is polynomial function of k. 

Consider a given deformation 0 6 9Jt fc (q, q'). It can be decomposed into a sum of 
homogeneous deformations: = ip 0 + ipi + ■ ■ ■ + r /> k . Then = fpo + l/h is an affine 
deformation. More schematically, we can write </> = ^ where L = C 2 + • • • + V’fc- 

Let q'^ = (q\ Al • • • , q' pA ) be the projected points after the deformation Note that 
they are not observed on the image since they are theoretically produced by the affine part 
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of the actual deformation. We have of course: ip a G 9J1, (q. q'/J. For any other affine 
deformation ip' A G 97ti(q, q'^), the deformation o’ — fi>' A + X lies in 9Jt fc (q, q'). Therefore 
for a generic situation the following holds: 

dim (07lfc(q, q')) > dim (9Jli(q, q' A )) = 4 
We are now in a position to formulate and prove the following theorem: 

Theorem 8. At least three images are required in order to recover the structure and de¬ 
formation of a point set from a single calibrated camera. 

Proof Now let I C x A 2n be the variety defined by the system of equations, where d 
denotes the degree of the selected model. Let 7Ti : X —* ‘X,/ and 7 t 2 : X — * A 2n be the 
canonical projections. By If20l . pp. 78-79, it is always true that dim(A) > dim(7Ti(A)), 
where Tti(X) denotes the Zariski closure of Ti\ [X). With the previous notations, we have: 

= 9Kd(q, q'). Hence dim(A) > 4 > 0. Thus one cannot recover the deformation 
(and the structure) from two images only. □ 

Therefore a third image must be considered. Following results from previous sections, 
in our setting, we considered the case where the scene undergoes the same deformation 
twice. Therefore one must add the following equations: 

q'i A = 0 (20) 

A simple counting procedure provides us the minimum number of points necessary to 
get a finite set of solutions. 

Proposition 1. Let d be the degree of the deformation and p be the dimension of the 
deformation space X,/. Under the assumption that the same deformation is repeated twice, 
at least p/3 input points are necessary to get a finite set of solutions. 

Proof. Equations (1 191201) provide 4 independent equations per point. The localization 
equations (fl8l) are n independent equations. There are p + 2// unknowns. Therefore for a 
finite set of solutions, it is necessary that An + n > p + 2 n. Thus n > p/3. □ 

In order to illustrate our considerations, we directly and simultaneously compute the 
structure and the deformation from equations (fl8l) . (fl9l) and (l20l) . More precisely, a set of 
points has been deformed with the following deformation: 

/X\ fl/AX 2 + XY + X + Y + Z+ 1\ 

$ Iy = 1/41' 2 + YZ + X + Y + Z + 1 

\z) \l/AZ 2 + XZ + X + Y + Z + lJ 

Both the deformation and the original set of points have been computed and as ex¬ 
pected there is a single (real) solution. The computations have been performed using the 
minimal number of points presented in proposition|T|and were done with Maple. 
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4 Conclusion 


We introduced a new problem in multiple-view geometry, i.e. the recovery of structure and 
deformation from a single perspective camera, where the deformation is either an affine 
or polynomial morphism of A 3 and the camera is either calibrated or not. We showed 
several theoretical results and provided experiments in simple settings as a proof of con¬ 
cept. This paves the way for further theoretical and practical research about deformable 
configurations of points viewed from a monocular sequence. 
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